{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a CNN for DPU compilation\n",
    "\n",
    "In this notebook we show how to train a simple Convolutional Neural Network (CNN)\n",
    "on the MNIST handwritten digit dataset for deployment on the DPU. We will cover:\n",
    "\n",
    "* Loading and pre-processing the imagenet dataset\n",
    "* Training a CNN with Keras and Tensorflow\n",
    "* Freezing the trained model\n",
    "* Quantizing and evaluating the quantized model\n",
    "* Compiling for DPU using the Vitis AI compiler\n",
    "\n",
    "**Note**:\n",
    "* This notebook should be run on a proper X86 machine; it has to support\n",
    "AVX2 and FMA instructions sets. To check:\n",
    "```shell\n",
    "grep avx2 /proc/cpuinfo\n",
    "grep fma /proc/cpuinfo\n",
    "```\n",
    "* Please make sure the following has been done in terminal before you start this notebook:\n",
    "```shell\n",
    "conda activate vitis-ai-tensorflow\n",
    "yes | pip install matplotlib keras==2.2.5\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)\n",
    "import keras\n",
    "from keras.layers import Dense, Conv2D, InputLayer, Flatten, MaxPool2D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "The MNIST dataset comes with 60k training and 10k test examples that are\n",
    "28x28 grayscale images, along with their labels which are just the \n",
    "corresponding digits saved as integers. We can use the `keras.datasets`\n",
    "utility to load the MNIST dataset straight into our Jupyter environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data: (60000, 28, 28), (60000,)\n",
      "Test data: (10000, 28, 28), (10000,)\n"
     ]
    }
   ],
   "source": [
    "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
    "\n",
    "print('Training data: {}, {}'.format(x_train.shape, y_train.shape))\n",
    "print('Test data: {}, {}'.format(x_test.shape, y_test.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr0AAACTCAYAAACDHaG3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAbNklEQVR4nO3deZSU1Z3G8ecnBtEQRJQQAgHUIB71uERQQhw1AdQYjfvCiSzGiZ4YleQoAxo0ZCKGuM3gGg0CihzRIyguMegoaozKgAYTQBA1AwERWUTWhKB3/qji9d4Xqrv67Vreeuv7OaePv9u3q95b3Y/N7apb95pzTgAAAECW7VLtAQAAAADlxqQXAAAAmcekFwAAAJnHpBcAAACZx6QXAAAAmcekFwAAAJnHpDfGzF40s3+v9G1R28gNkiA3aCoygyTITU5mJ71m9n9m1q/a4yjEzIaY2admttH7OL7a46p3ac+NJJnZz8zsQzNbb2bjzWy3ao+p3tVCbrYzs+fNzJnZrtUeSz1Le2bM7BAzm2Fmq82MDf1TogZys5uZ/ZeZfWBmH5vZXWb2hWqPa7vMTnprxGvOudbex4vVHhDSzcxOlDRCUl9JXSXtJ+mXVR0UaoaZ/UBSav4BQqr9S9Ijki6q9kBQU0ZI6inpEEkHSPqGpJFVHZGn7ia9ZraXmT1lZqvyf4U8ZWadY1+2v5n9b/6ZtOlm1s67fW8ze9XM1pnZWzw7Wx9SlJvBku5zzs13zn0s6VeShiS8L5RZinIjM9tT0i8k/UfS+0D5pSUzzrlFzrn7JM1vxsNBhaQlN5JOlXSbc26tc26VpNsk/TDhfZVc3U16lXvME5R7lqyLpC2S7oh9zSDlfkgdJW1T7ocmM+sk6WlJ10tqJ+kqSVPNrH38ImbWJR+eLg2M5Yj8S0fvmNm1vNyYamnJzcGS3vLab0nqYGZ7J3xcKK+05EaSbpB0t6QPm/OAUHZpygxqR5pyY7G6c/6P7qqru0mvc26Nc26qc26zc26DpNGSjot92STn3Dzn3CZJ10o618xaSLpA0u+dc793zn3mnHtO0hxJJ+/kOkudc22dc0sLDOVl5Z7+/7KksyQNkDSsJA8SJZei3LSW9InX3l5/qRkPD2WSltyYWU9J35J0ewkfHsogLZlBbUlRbv4gaaiZtTezr0i6Iv/5PUrwMJut7ia9ZraHmd1jZkvMbL1yk8+2+R/8dn/36iXKrYHbR7m/oM7J/5WzzszWSTpGub+amsQ5975z7m/5gP1V0n9KOjvp40J5pSU3kjZKauO1t9cbEtwXyiwNuTGzXSTdJWmoc25bcx4Pyi8NmUHtSVFuRkv6s6S5kl6V9Lhy68NXJrivkqu7Sa+kKyX1kHS0c66NpGPzn/efjv+aV3dR7ge2WrnATMr/lbP944vOuTElGJeLjQHpkpbczJd0mNc+TNJK59yaBPeF8ktDbtoo98aSh83sQ0mz859fZmb/1sT7QvmlITOoPanIjXNui3PuMudcJ+fcfpLWSHrDOfdZkgdValmf9H7BzFp5H7sq9zLwFknr8ou4f7GT211gZgeZ2R7KPQP7qHPuU0kPSjrVzE40sxb5+zx+J4vFG2Vm3zWzDvn6QOVeapie8HGitFKbG0kPSLoof522yr0rdmKSB4mSS2tuPpH0VUmH5z+2v2R5pKRZTX+YKKG0ZkaW00pSy3y7lbE9YlqkOTedzOyr+fz0Vm5us7OxVEXWJ72/Vy4E2z9GSfpvSbsr99fN68qtP4mbpNxE4kNJrZRfk+Kc+7uk0yRdI2mVcn8dDdNOvo+WW+y90Qov9u4r6S9mtik/zmnKvdEE1Zfa3Djn/iDpRkkzJS1V7iWq1PxCqXOpzI3L+XD7R/6+pNwrBFuTPliURCozk9c1P6btuzdskbSoiY8P5ZHm3Oyv3LKGTZLulzTCOfdsgsdYFuYce04DAAAg27L+TC8AAADApBcAAADZx6QXAAAAmdesSa+ZnWRmi8zsXTMbUapBIdvIDZIgN0iC3CAJcpNNid/Ilt/w+B1J/SUtU27vxwHOuQWlGx6yhtwgCXKDJMgNkiA32bVrM257lKR3nXPvS5KZTVFuy4uCoTAztopIGedcpQ/EIDcZQG6QRNpzQ2ZSabVzrn2Fr0lualyh3zXNWd7QSeGRdsvynwuY2cVmNsfM5jTjWsgOcoMkyA2SaDQ3ZCb1llThmuQmo5rzTG9RnHP3SrpX4q8hFI/cIAlyg6YiM0iC3NSm5jzTu1zhOc6d858DGkJukAS5QRLkBkmQm4xqzqR3tqTuZravmbWUdL6kJ0ozLGQYuUES5AZJkBskQW4yKvHyBufcNjO7TNIMSS0kjXfOzW/kZqhz5AZJkBskQW6QBLnJrsRbliW6GOteUqcK76ZuMnKTPuQGSaQ9N2Qmld5wzvWs9iAaQm7Spxy7NwAAAAA1gUkvAAAAMo9JLwAAADKPSS8AAAAyj0kvAAAAMo9JLwAAADKv7McQA9i5I488MmhfdtllUT1o0KCg74EHHgjat99+e1S/+eabZRgdAADZwjO9AAAAyDwmvQAAAMg8Jr0AAADIPI4hLqBFixZBe8899yz6tv7azD322CPo69GjR9D+yU9+EtU333xz0DdgwICo/sc//hH0jRkzJmj/8pe/LHp8vrQfCyrVVm4acvjhhwftF154IWi3adOm6Pv65JNPonrvvfdu3sASIDe1r2/fvkF78uTJUX3ccccFfYsWLSrJNdOeGzIjjRw5Mmj7/7bsskv4PNnxxx8f1S+99FK5hsQxxGgyjiEGAABA3WLSCwAAgMxj0gsAAIDMy/w+vV26dInqli1bBn19+vQJ2sccc0xUt23bNug766yzSjKeZcuWBe3bbrstqs8444ygb8OGDVH91ltvBX1lXD+FEjrqqKOieurUqUFffJ24v77e/9lL0tatW4O2v463d+/eQZ+/b2/8dmjcscceG7Tja6Yfe+yxSg6nbHr16hW0Z8+eXaWRoJqGDBkStIcPHx60P/vss4K3reR7goBS4JleAAAAZB6TXgAAAGRe5pY3NLQtVFO2HSuV+EtD8e1gNm7cGNX+lkGStGLFiqj++OOPg75SbSGE5vO3pfvGN74R9D344INR3bFjx6Lvc/HixUH7xhtvDNpTpkyJ6j/96U9Bn5+xX//610VfEzn+NkyS1L1796Bdy8sb/C2n9t1336Cva9euUW2W6p3FUEL+z12SWrVqVaWRoBKOPvrooH3BBRdEdXyrwoMPPrjg/Vx11VVB+4MPPgja/nJR/99BSZo1a1Zxgy0DnukFAABA5jHpBQAAQOYx6QUAAEDmZW5N79KlS4P2mjVrorpUa3rj61HWrVsXtL/97W9HdXzLqEmTJpVkDEiPe+65J6r9o6ObI742uHXr1kHb37Iuvgb10EMPLckY6tWgQYOC9muvvValkZSev678Rz/6UdDnr7tbuHBhxcaEyuvXr19UX3755Q1+rZ+FU045JehbuXJlaQeGsjjvvPOieuzYsUHfPvvsE9Xxtfwvvvhi0G7fvn1U33TTTQ1e078v/3aSdP755zc84DLimV4AAABkHpNeAAAAZF7mljesXbs2aA8bNiyq4y/N/PnPfw7a/ulocXPnzo3q/v37B32bNm0K2v42H0OHDm1kxKg1Rx55ZND+3ve+F9UNbfUUP0XvySefDNo333xzVMe3f4ln1d/C7jvf+U7Qx3ZTzeNv65U148aNK9gX3yYP2eFvHyVJEyZMiOrGlv35L2MvWbKktANDSey6aziV69mzZ9D+3e9+F9X+FpuS9PLLL0f1r371q6DvlVdeCdq77bZbVD/yyCNB3wknnFBwfHPmzCnYV2nZ/e0OAAAA5DHpBQAAQOY1Ouk1s/Fm9pGZzfM+187MnjOzxfn/7lXeYaLWkBskQW6QBLlBEuSm/phzruEvMDtW0kZJDzjnDsl/7kZJa51zY8xshKS9nHPDG72YWcMXK7M2bdoE7Q0bNgRtf+upiy66KOjzj+p76KGHyjC66nDOlWUBaJZy09DR1tKOufI988wzUR3fzix+5KO/1Vh87eWqVasKXuPTTz8N2ps3by54jTfffLPg/TRF1nLjf+/jW5RNmzYtaA8cOLDYu02dV199Nap79+4d9PXp0yeqX3/99bJcP+25qfbvmnLx13RK0g9/+MOCXxvfpqpv377lGFJTvOGc69n4lzVdVnIzZMiQoN3Q2v3nnnsuaPvbma1fv77B6/jzoIkTJzb4tcuXL4/q+Brjhv49K5VCv2safabXOfeypLWxT58m6f58fb+k05s1OmQOuUES5AZJkBskQW7qT9LdGzo451bk6w8ldSj0hWZ2saSLE14H2UJukAS5QRJF5YbMIIbcZFiztyxzzrmGntp3zt0r6V6p+i8BID3IDZIgN0iiodyQGRRCbrIn6aR3pZl1dM6tMLOOkj4q5aDKpbH1Kp988knBPv/Izocffjjo++yzz5o3sPpRM7k54IADotrf61nacV/L1atXR/WKFSuCvvvvvz+qN27cGPQ9/fTTDbaT2n333aP6yiuvDPp+8IMflOQaFVb23Jx88slR7X//al2HDuGTVPvuu2/Br/XX4GVEzfy+KTX/aFlpxzW8/r9Z69atC/quv/768g2sNtREbvw9da+55pqgL/5erbvuuiuqR44cGfQ1Ni/y/fznPy/6a6+44oqorsQa3mIl3bLsCUmD8/VgSdNLMxxkHLlBEuQGSZAbJEFuMqyYLcsekvSapB5mtszMLpI0RlJ/M1ssqV++DUTIDZIgN0iC3CAJclN/Gl3e4JwbUKCr6vuYlNqoUaOiOn7UrL/1U79+/YK+Z599tqzjqkW1lhv/eEUpPBLYf+lb2nGru0GDBkV1/LjFar9U3qVLl6pev6mqlZsePXoU7Js/f345L11Wfo6lcLnDO++8E/TFc11Lau33TTl069YtqqdOnVr07W6//fagPXPmzFINKfVqKTfXXXdd0PaXNGzdujXomzFjRtAePvzzHde2bNlS8BqtWrUK2vGjhf1/T+LH3ceXxUyfns4nyDmRDQAAAJnHpBcAAACZx6QXAAAAmdfsfXqzZNOmTVHtb1Emhce3xo90jK+B8td13nnnnUFfY8c+ozqOOOKIoB1fx+s77bTTgvZLL71UljEhHWbPnl3tIQTix16fdNJJUe0fEyrtuCbP5295JO24dRVqi58D/1jtnXn++eejeuzYsWUbE5qnbdu2UX3ppZcGff5cIr6G9/TTiz9E7utf/3pUT548OeiLv7fJ9+ijjwbtG2+8sehrVhPP9AIAACDzmPQCAAAg81jeUMB7770XtIcMGRLVEyZMCPoGDhxYsP3FL34x6HvggQeCdvwEL1THrbfeGrT97VjiyxfStpxhl13Cv105IbC02rVrl+h2hx12WNCOb/Hjb33YuXPnoK9ly5ZRHT9FL/7z9rcgmjVrVtD3z3/+M2jvuuvnv/LfeOONgmNH+sVfwh4zpvB2sq+88krQHjx4cFQ3dBIpqsv/PRA/Zc/nn34mSV/+8peD9oUXXhjV3//+94O+Qw45JKpbt24d9MWXY/rtBx98MOjzl4emGc/0AgAAIPOY9AIAACDzmPQCAAAg81jTW6THHnssqhcvXhz0xdeD9u37+QmGN9xwQ9DXtWvXoD169OioXr58ebPHieKdcsopUX344YcHff7apSeeeKJiY0oivobXH/vcuXMrPZya5K+Lja9j++1vfxu0/eM/GxLfNiq+pnfbtm1RvXnz5qBvwYIFUT1+/PigL37Utb/GfOXKlUHfsmXLgrZ/LPbChQsLjh3plPSo4ffffz9ox3OCdPKPF161alXQ1759+6j+29/+FvQ1ZWvUDz74IKrXr18f9HXs2DFor169OqqffPLJoq+RJjzTCwAAgMxj0gsAAIDMY9ILAACAzGNNbwLz5s0L2ueee27QPvXUU6M6vqfvJZdcErS7d+8e1f379y/VEFEEf32jvx+iJH300UdR/fDDD1dsTIXstttuQXvUqFEFv/aFF16I6quvvrpcQ8oU/4jPJUuWBH19+vRJdJ9Lly4N2o8//njQfvvtt6P69ddfT3SNuIsvvjho++v+pB3XdqK2DB8+PKqbsh93Q3v4Ir38o8Hj+zI/9dRTUR3fSzx+zsD06dOjeuLEiUHf2rVro3rKlClBX3xNb7y/FvFMLwAAADKPSS8AAAAyj+UNJeC/BCFJkyZNiupx48YFff4xoJJ07LHHRvXxxx8f9L344oulGSCazD++tRpHRceXM4wcOTJoDxs2LKrj21LdcsstUb1x48YyjC7bfvOb31R7CIn52yXuTFO2uUL1xbdSPOGEE4q6nf9ytiQtWrSoZGNCdcSPGI8vXUrKn4Mcd9xxQV98CU0WlkfxTC8AAAAyj0kvAAAAMo9JLwAAADKPNb0JxI8XPfvss4N2r169ojq+hjfOP2705ZdfLsHoUArVOHrYX7/nr9mVpPPOOy9o+2v2zjrrrPIODJnhH6eO9Hv22WeD9l577VXwa/1t74YMGVKuISFj/K07GzrSXmLLMgAAAKAmMOkFAABA5jHpBQAAQOaxpreAHj16BO3LLrssqs8888yg7ytf+UrR9/vpp58GbX8P2KYcK4nmM7Od1lJ45OPQoUPLcv2f/exnQfvaa6+N6j333DPomzx5ctAeNGhQWcYEID323nvvoN3QvxF33XVXVLM/N4o1Y8aMag+honimFwAAAJnX6KTXzL5mZjPNbIGZzTezofnPtzOz58xscf6/hd9WirpDbpAEuUFTkRkkQW7qUzHLG7ZJutI596aZfUnSG2b2nKQhkp53zo0xsxGSRkgaXr6hll58WcKAAQOi2l/OIEndunVLdI05c+YE7dGjRwftamyNVSGpz42/HUt8axY/G7fddlvQN378+KC9Zs2aqO7du3fQN3DgwKg+7LDDgr7OnTsH7aVLl0Z1/CUn/6XLjEt9bmpJfNnOAQccENX+Flc1LjOZmTBhQtDeZZfiX4x99dVXSz2crMtMbprjxBNPrPYQKqrR/6Occyucc2/m6w2S3pbUSdJpku7Pf9n9kk7f+T2gHpEbJEFu0FRkBkmQm/rUpDeymVk3SUdImiWpg3Nu+7uwPpTUocBtLpZ0cfIhotaRGyRBbtBUZAZJkJv6UfRrJ2bWWtJUST91zq33+1zutWG3s9s55+51zvV0zvVs1khRk8gNkiA3aCoygyTITX0p6pleM/uCcqGY7Jyblv/0SjPr6JxbYWYdJX1UrkE2R4cOn/+RdtBBBwV9d9xxR9A+8MADE11j1qxZQfumm26Kav+4WKm+tiWr5dy0aNEiqi+99NKgL37s7/r1n/+e7N69e9HXiK/BmzlzZlRfd911Rd9P1tRybtImvla9KWtEa0ktZ8Y/frxfv35BX/zfi61bt0b1nXfeGfStXLmyDKPLtlrOTanst99+1R5CRRWze4NJuk/S2865W72uJyQNzteDJU2P3xb1i9wgCXKDpiIzSILc1Kdinun9lqSBkv5qZnPzn7tG0hhJj5jZRZKWSDq3PENEjSI3SILcoKnIDJIgN3Wo0Umvc+4VSVagu29ph5NMu3btovqee+4J+vyXjprzNL7/UvQtt9wS9MW3l9qyZUvi62RFLeTmtddei+rZs2cHfb169Sp4u/hWd/4Smjh/O7MpU6YEfeU66a2W1UJuatk3v/nNqJ44cWL1BlJCtZ6Ztm3bRnVjp3suX748qq+66qqyjake1HpuSuWPf/xjVMeXP2VxOWY2F3gBAAAAHia9AAAAyDwmvQAAAMi8Jh1OUS1HH3100B42bFjQPuqoo6K6U6dOia+zefPmqI4fPXvDDTdE9aZNmxJfA+mxbNmyqD7zzDODvksuuSSqR44cWfR9jh07NmjffffdUf3uu+82dYhAs8SPIQYA37x586J68eLFQV/8fVD7779/VK9ataq8AysTnukFAABA5jHpBQAAQOYx6QUAAEDm1cSa3jPOOKPBdkMWLFgQ1U899VTQt23btqDt77+7bt26pgwRNW7FihVBe9SoUTutgTR75plngvY555xTpZGgWAsXLozq+NHkxxxzTKWHgzrmv3dJksaNGxe0R48eHdWXX3550OfPtdKMZ3oBAACQeUx6AQAAkHnmnKvcxcwqdzEUxTmX+j2NyE36kBskkfbckJlUesM517Pag2hIVnLTpk2boP3II48E7X79+kX1tGnTgr4LL7wwaFd7a9dCv2t4phcAAACZx6QXAAAAmcekFwAAAJnHmt46l/Y1dhK5SSNygyTSnhsyk0qs6a2S+Bpff8uyH//4x0HfoYceGrSrvYUZa3oBAABQt5j0AgAAIPNY3lDn0v5yo0Ru0ojcIIm054bMpBLLG9BkLG8AAABA3WLSCwAAgMxj0gsAAIDM27XC11staYmkffJ1GqRpLFJlx9O1QtdprtWSNql+f07FIDc7IjeNIzehNP4bJdX3eMhNcvU6noKZqegb2aKLms1Jy8L0NI1FSt940iJt3xfGUxvS9n1hPLUhbd8XxlMb0vZ9YTw7YnkDAAAAMo9JLwAAADKvWpPee6t03Z1J01ik9I0nLdL2fWE8tSFt3xfGUxvS9n1hPLUhbd8XxhNTlTW9AAAAQCWxvAEAAACZV9FJr5mdZGaLzOxdMxtRyWvnrz/ezD4ys3ne59qZ2XNmtjj/370qOJ6vmdlMM1tgZvPNbGi1x5RG5CYYC5kpErkJxkJuikRugrGQmyJUOzP5MZCbIlRs0mtmLSTdKem7kg6SNMDMDqrU9fMmSjop9rkRkp53znWX9Hy+XSnbJF3pnDtIUm9JP8l/T6o5plQhNzsgM0UgNzsgN0UgNzsgN41ISWYkclMc51xFPiR9U9IMr321pKsrdX3vut0kzfPaiyR1zNcdJS2q9Ji8sUyX1D9NY6r2B7khM+SG3JCbdPycyE16M0Nuivuo5PKGTpL+7rWX5T9XbR2ccyvy9YeSOlRjEGbWTdIRkmalZUwpQW4KIDMNIjcFkJsGkZsCyE1Bac2MlIKfU9pywxvZPC7350fFt7Mws9aSpkr6qXNufRrGhOJV42dEZmofuUES5AZJkJucSk56l0v6mtfunP9cta00s46SlP/vR5W8uJl9QblQTHbOTUvDmFKG3MSQmaKQmxhyUxRyE0NuGpXWzEjkZgeVnPTOltTdzPY1s5aSzpf0RAWvX8gTkgbn68HKrT2pCDMzSfdJets5d2saxpRC5MZDZopGbjzkpmjkxkNuipLWzEjkZkcVXsx8sqR3JL0n6eeVXsAs6SFJKyT9S7l1NxdJ2lu5dxEulvQ/ktpVcDzHKPf0/l8kzc1/nFzNMaXxg9yQGXJDbsgNuUnrR7UzQ26K/+BENgAAAGQeb2QDAABA5jHpBQAAQOYx6QUAAEDmMekFAABA5jHpBQAAQOYx6QUAAEDmMekFAABA5jHpBQAAQOb9P7Bg60W5zyq4AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axs = plt.subplots(1, 5, figsize=(10, 10))\n",
    "plt.tight_layout()\n",
    "\n",
    "for i in range(5):\n",
    "    axs[i].imshow(x_train[i], 'gray')\n",
    "    axs[i].set_title('Label: {}'.format(y_train[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will normalize the training and test images. We also need to add\n",
    "color channel. All these pre-processing steps are required by Conv2D layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "x_train = np.expand_dims(x_train, axis=3)\n",
    "x_test = np.expand_dims(x_test, axis=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "\n",
    "Create a sequential model by passing a list of layers. Because the MNIST\n",
    "dataset is not difficult, we can use a very simple network with a single \n",
    "convolutional layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    InputLayer(input_shape=(28, 28, 1), name='input_data'),\n",
    "    Conv2D(32, 3, activation='relu'),\n",
    "    MaxPool2D(pool_size=(2,2)),\n",
    "    Flatten(),\n",
    "    Dense(128, activation='relu'),\n",
    "    Dense(10, activation='softmax', name='output_logits')\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 5408)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 128)               692352    \n",
      "_________________________________________________________________\n",
      "output_logits (Dense)        (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 693,962\n",
      "Trainable params: 693,962\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Configure the model for training: choose desired optimizer, loss function\n",
    "and metrics to observe over the training period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss=\"sparse_categorical_crossentropy\", \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "60000/60000 [==============================] - 9s 153us/step - loss: 0.1553 - acc: 0.9536\n",
      "Epoch 2/5\n",
      "60000/60000 [==============================] - 9s 151us/step - loss: 0.0520 - acc: 0.9837\n",
      "Epoch 3/5\n",
      "60000/60000 [==============================] - 9s 148us/step - loss: 0.0318 - acc: 0.9903\n",
      "Epoch 4/5\n",
      "60000/60000 [==============================] - 9s 147us/step - loss: 0.0213 - acc: 0.9933\n",
      "Epoch 5/5\n",
      "60000/60000 [==============================] - 9s 146us/step - loss: 0.0138 - acc: 0.9956\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect the training results by plotting the collected data in the\n",
    "`history` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAEWCAYAAACt0rvRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdeXyV5Z3//9cnG4EkJCwJS0LIBrKDGhAXdlu1rdK6VK1atba2KrYznc5M++1vbGun03XaGQXrhlu1VWtby1SttbIrIrgAggvZIGGRNSEBAlk+vz/OIY0xQICc3CfJ+/l4nAfn3Pd1n/M+KHc+ue7rvi5zd0RERERE5NTFBB1ARERERKSrUHEtIiIiItJOVFyLiIiIiLQTFdciIiIiIu1ExbWIiIiISDtRcS0iIiIi0k5UXEuXZmYvmNn17d32BDNMN7OK9n5fEZFoFA3nXZEgmea5lmhjZjXNXvYCDgEN4ddfdfcnOj7VyTOz6cDj7p4VdBYRkdZ0tfOuSJDigg4g0pK7Jx95bmZlwJfd/e8t25lZnLvXd2Q2EZGuSOfd9qW/p+5Nw0Kk0zgyvMLM/t3MtgMPm1kfM/uLme00s73h51nNjllsZl8OP7/BzJab2S/CbUvN7KKTbJtrZkvNrNrM/m5m88zs8TZ+j5Hhz6o0s/VmdkmzfZ8ysw3h991iZt8Kb+8f/m6VZrbHzJaZmf79ikhEddbzbhsy9jWzh81sa3j/s832zTazt81sn5kVm9mF4e1lZnZ+s3bfP/L5ZpZjZm5mN5nZZmBhePvvzWy7mVWFs49udnxPM/tvM9sU3r88vO05M7u9xfdZa2afO9H/fhIM/XCWzmYg0BcYCtxM6P/hh8Ovs4GDwNxjHH8W8D7QH/gZMN/M7CTa/hZ4HegHfB+4ri3hzSwe+D/gb0AGcDvwhJmdFm4yn9Al2BRgDOETNPAvQAWQDgwA/h+gMV0i0hE643n3eBl/Q2j4y2hC5+JfAZjZJOAx4F+BNGAqUHaMz2lpGjASuCD8+gVgWPgz3gSaD6/5BXAmcA6hv99/AxqBR4FrjzQys/FAJvDcCeSQAKm4ls6mEfieux9y94Puvtvd/+DuB9y9GvgRoZPb0Wxy9wfcvYHQCWwQoWK1zW3NLBuYCNzh7ofdfTmwoI35JwPJwE/Cxy4E/gJcHd5fB4wys97uvtfd32y2fRAw1N3r3H2Z64YJEekYne68e6yMZjYIuAj4Wvg8W+fuS8KH3gQ85O4vuXuju29x9/fa9tcEwPfdfb+7HwzneMjdq939EKFfCMabWWr4yuOXgG+EP6PB3V8Nt1sADDezYeH3vA54yt0Pn0AOCZCKa+lsdrp77ZEXZtbLzO4LX1bbBywF0sws9ijHbz/yxN0PhJ8mn2DbwcCeZtsAytuYfzBQ7u6NzbZtItQrAXAZ8Clgk5ktMbOzw9t/DhQBfzOzEjP7dhs/T0TkVHW68+5xMg4Jv9feVg4dAhQf7X3boCmTmcWa2U/CQ0v28Y8e8P7hR2JrnxX+u34KuDZchF9NqKddOgkV19LZtOyt/RfgNOAsd+9N6BIewNEuObaHbUBfM+vVbNuQNh67FRjSYrx0NrAFwN1XuftsQpcQnwWeDm+vdvd/cfc84BLgm2Y26xS/h4hIW3TG8+6xMpaH3yutlePKgfyjvOd+QkNJjhjYSpvmf1dfAGYD5wOpQE6zDLuA2mN81qPANcAs4IC7rzhKO4lCKq6ls0shNJau0sz6At+L9Ae6+yZgNfB9M0sI9y5f3MbDVwIHgH8zs3gLTdN3MfBk+L2uMbNUd68D9hG6HIuZfcbMCsJjD6sITZHV2PpHiIhEVGc47x41o7tvIzQW+p7wjY/xZnak+J4P3Ghms8wsxswyzWxEeN/bwFXh9oXA5ceJnUJoSsPdhIry/2qWoRF4CPilmQ0O93KfbWY9wvtXEDrH/zfqte50VFxLZ/c/QE9CvQCvAX/toM+9Bjib0EnzPwldwjt0vIPCY+YuJjTebxdwD/DFZmP6rgPKwpcQvxb+HAjdEPN3oAZYAdzj7ova7duIiLRdZzjvHi/jdYTuZXkP2AH8E4C7vw7cSOgGxypgCaGbIgH+g1BP817gB4RusDyWxwgN+9sCbAjnaO5bwDpgFbAH+CkfrcseA8YCbZqJSqKHFpERaQdm9hTwnrtHvAdHRES6/nnXzL4I3Ozu5wWdRU6Meq5FToKZTTSz/PBlwwsJjat79njHiYjIyelO593w2PJbgfuDziInTis0ipycgcAfCc23WgHc4u5vBRtJRKRL6xbnXTO7gND3/DvHH3oiUUjDQkRERERE2omGhYiIiIiItJMuMyykf//+npOTE3QMEZGT8sYbb+xy9/Sgc3QknbdFpLM61jk7osV1+IaD/wVigQfd/Sct9k8lNF3OOOAqd3+m2b5s4EFCk8Q78Cl3LzvaZ+Xk5LB69ep2/w4iIh3BzDYFnaGj6bwtIp3Vsc7ZERsWEl5idB6h+XxHAVeb2agWzTYDN9D6gP3HgJ+7+0hgEqF5KEVEREREolYke64nAUXuXgJgZk8SmjZnw5EGR3qizewjK82Fi/A4d38p3K4mgjlFRERERNpFJG9ozATKm72uCG9ri+GEliz9o5m9ZWY/D/eEf4SZ3Wxmq81s9c6dO9shsoiIiIjIyYvW2ULigCmElgadCOQRGj7yEe5+v7sXunthenq3ug9IRERERKJQJIvrLYRuRjwiK7ytLSqAt929xN3rCa3AdEY75xMRERERaVeRLK5XAcPMLNfMEoCrgAUncGyamR3pjp5Js7HaIiIiIiLRKGLFdbjHeQ7wIvAu8LS7rzezO83sEgAzm2hmFcAVwH1mtj58bAOhISEvm9k6wIAHIpVVRERERKQ9RHSea3d/Hni+xbY7mj1fRWi4SGvHvkRo/uuIWVdRxQvvbOPfLhwRyY8RERERkSjg7uyqOUzxzhqKdtSwu+Yw3zh/WLt+RpdZofFkvLFpD/csLmb6aRlMyu0bdBwRERERaQf1DY1s3nOA4p37Kd5ZQ/GOGorCf+6rrW9ql9Ijjltn5BMf236DObp1cX3VpGzmLipi7qIiHsudFHQcERERETkBNYfqKdlZ09QTXbwjVEyX7d5PXYM3tctI6UF+ejKXTBhMQXoy+RnJ5KcnM7B3IjEx1q6ZunVxnRgfy5fOy+Vnf32ftRWVjMtKCzqSiIiIiDTj7uyoPvSR3ufinfsp2lHD9n21Te1iY4yh/XqRn57M+aMGkJ+eTH56EnnpyaT2jO+wvN26uAa4bvJQ7l1czLxFRdx3XWHQcURERES6pbqGRjbt3k9RuPe5uFkhXXPoH0M5knvEkZ+exDkF/cIFdDIFGUlk900iIS74JVy6fXGdkhjPDefkcNfCIjZ+WM2wASlBRxIRERHpsvbV1jUVzU3DOXbWsHn3Aeob/zGUY1BqIvnpyVx2Rib5GclNwzkyUnpg1r5DOdpTty+uAW48N5cHl5dyz+JifnXlhKDjiIiIiHRq7s62qtqPFM9HxkPvqD7U1C4+1sjpl8TwjBQuGjOQgvBY6Lz0ZJJ7dM4ytXOmbmd9khK45qxsHnqljH8+fzjZ/XoFHUlEREQk6h2qb2DT7gPhmwnDNxburKFk534OHG5oapeSGEdBRjJTh6c3FdD56Ulk9+1FXDvO1BENVFyHfXlKHo++uolfLynmx5eODTqOiIiISNSoPHD4I73PTUM59hyg2UgOMtN6kp+RzMScvs3GQyfTPzkhqodytCcV12EDeidyRWEWv19dwTdmDWNgamLQkUREREQ6TGOjs6XyYLPiOVRIl+ysYVfN4aZ2CXEx5PVPYvTgVC4ZP7hpWru89CR6Jai01N9AM1+bls+Tq8p5YFkJ//GZUUHHEREREWl3tXUNlO7a/4+x0OFp7Up31VBb19jULq1XPAXpycwaMSA0lCMjifz0ZLL69CK2neeG7kpUXDczpG8vZk8YzG9Xbua2GQX0TUoIOpKIiIjISXEP9USvrahibUUV723fR/HOGir2HsTDQznMIKtPT/LTkzk3v19TL3R+ehL9knsE+wU6KRXXLdw6PZ8/vbWFh5aX8q0LTgs6joiIiEib7Kw+xNqKStZUVLG2opJ1FVXs3h8azhEfaxRkpDBhSB8uOyOraSx0bv8kEuNjA07etai4bqEgI4ULRw/k0RVl3Dwtj96JHbeij4iIiEhbVB2oY92WKtZUVLK2opK1FVVsqwqtVhhjMCwjhRkjMhiflcq4rDRGDEqhR5yK6I6g4roVt80o4IV3tvObFZu4bUZB0HFERESkGztwuJ53tuxrKqLXVlRStvtA0/6cfr2YmNOXceFCevTg3iR10jmiuwL9zbdiTGYq009L56HlpXzp3Fx6Jug3PREREYm8Q/UNvLetulkhXcXGHdVN090NSk1kXFYqVxQOYXxWGmMzU0ntpavs0UTF9VHMmVHA5feu4Hevb+ZL5+UGHUdERES6mPqGRop21rC2/MjwjtBNh3UNoUq6b1IC47JSuWDMQMZnpTI2K5WMFE0VHO1UXB9FYU5fJuX25f6lJVwzOVvjlEREROSkNTY6Zbv3h8ZJl4eGdqzfuo+DdaFVDFN6xDE2K5WbzssLD+9IJTOtZ7dZeKUrUXF9DHNmFPDFh17nj29u4epJ2UHHERERkU7A3dlaVcva8krWbqlqGuJRXVsPQGJ8DKMHp3LVpPDQjqxUcvslEaO5o7uEiBbXZnYh8L9ALPCgu/+kxf6pwP8A44Cr3P2ZFvt7AxuAZ919TiSztmbKsP6My0rl3iXFXHFmFnGxMR0dQURERKLcrprwFHjlVawLF9NHVjSMjzVGDOzNxeMHN83cMSwjWTVFFxax4trMYoF5wCeACmCVmS1w9w3Nmm0GbgC+dZS3+SGwNFIZj8fMuG1GAV/9zRs8t24bsydkBhVFREREokDVwTreOTIFXriY3lJ5EAgtyDIsI5npp2U0zdwxYmCK5pHuZiLZcz0JKHL3EgAzexKYTagnGgB3Lwvva2x5sJmdCQwA/goURjDnMX1i5ACGD0hm3qIiLh43WJdsREREuokDh+tZv3Vf0/R3ayuqKN21v2n/0H69OGNoH248N0dT4EmTSP4fkAmUN3tdAZzVlgPNLAb4b+Ba4Pz2j9Z2MTHGrdML+Ken3ualdz/kgtEDg4wjIiIiEXC4vpH3tu9jTUUV68KF9AcffnQKvLGZqVx+ZhbjslIZm5lKWq+EYENLVIrWX69uBZ5394pj3SVrZjcDNwNkZ0fuhsPPjBvEL1/6gHsWFfHJUQN0566IiEgn1tDoFO2oaVrdcF1FFe9uq+ZwQ+hC+pEp8D45eiDjMlMZN0RT4EnbRbK43gIMafY6K7ytLc4GppjZrUAykGBmNe7+7eaN3P1+4H6AwsJCP/XIrYuLjeGW6fl854/rWF60iynD0iP1USIiItKO3J1Nuw80zSO9tqKSd7Z8dAq8MZmp3HheTtOiLFl9NAWenLxIFtergGFmlkuoqL4K+EJbDnT3a448N7MbgMKWhXVHu/SMTP737xuZu7BIxbWIdEltmOFpKPAQkA7sAa5194rwvp8Cnw43/aG7P9VhwUWaaWx0XinexYri3U3F9L4WU+BdOXEI44eEbjjUFHjS3iJWXLt7vZnNAV4kdKJ+yN3Xm9mdwGp3X2BmE4E/AX2Ai83sB+4+OlKZTkWPuFi+MjWPH/5lA6vL9lCY0zfoSCIi7aaNMzz9AnjM3R81s5nAj4HrzOzTwBnABKAHsNjMXnD3fR37LaQ7q61r4Nm3tvDg8lKKdtQQF2OMGJTCZ8YPDg3tyEpj+ABNgSeRF9Ex1+7+PPB8i213NHu+itBwkWO9xyPAIxGId8KunjSEeYuKmLuoiEdunBR0HBGR9nTcGZ6AUcA3w88XAc82277U3euBejNbC1wIPN0RwaV727P/ML9ZsYnfvFbGrprDjBrUm19dOZ6LxgzSFHgSiGi9oTEq9UqI46bzcvn5i+/zzpYqxmSmBh1JRKS9tGWGpzXApYSGjnwOSDGzfuHt3zOz/wZ6ATP4aFHepKNuRJeur3hnDfOXl/KHNyo4VN/IjNPS+cqUPM7O76fx0hIoFdcn6Lqzh3LvkmLuWVzEPdecGXQcEZGO9C1gbvhemKWE7qdpcPe/hYf5vQrsBFYADa29QUfdiC5dk7uzsnQPDy4r4e/v7iAhLoZLT8/kpvNyGTYgJeh4IoCK6xPWOzGe68/OYd7iIop2VFOQoX/MItIlHHeGJ3ffSqjnGjNLBi5z98rwvh8BPwrv+y3wQQdklm6irqGR59dt48FlpazbUkXfpAS+PmsY100eSnpKj6DjiXyEiuuTcOO5OcxfXso9i4v55ecnBB1HRKQ9HHeGJzPrD+xx90bgO4RmDjlyM2Sau+82s3HAOOBvHRleuqbq2jqefL2cR14tY0vlQfL6J/Gjz43hsjOyNJ5aopaK65PQL7kHV0/K5tEVZfzz+cMZ0rdX0JFERE5JW2Z4AqYDPzYzJzQs5Lbw4fHAsvA4132Epuir7+jvIF3HlsqDPLy8lCdXlVNzqJ6zcvvyg0tGM3NEhqbNk6in4vok3Tw1j8df28R9S4v5z8+ODTqOiMgpa8MMT88Az7RyXC2hGUNETsnaikoeWFbK8+u2AfDpsYP48pRcxmWlBZxMpO1UXJ+kgamJXHZmFk+vruDrM4eR0VvLooqIiJyoxkZn4Xs7uH9ZCa+X7iG5Rxw3npPDjeflkpnWM+h4IidMxfUp+Nq0PJ5atZkHlpXw3U+r00ZERKStausa+MObFcxfXkrJzv0MTk3ku58ayZWThtA7MT7oeCInTcX1KRjaL4lLxg/miZWbuXV6AX2SEoKOJCIiEtV21RzisRWbePy1TezZf5hxWancdfXpXDRmIPFaPVG6ABXXp+jWGQU8+/ZWHn61jG9+YnjQcURERKLSxg+rmb+8lD++tYXD9Y2cPzKDr0zJY1JuXy36Il2KiutTNHxACheMHsAjr5TylSm5pOhSloiICBBa9GVF8W4eWFbCovd30iMuhsvPzOKm83LJT08OOp5IRKi4bge3zSjgxfUf8vhrm7llen7QcURERAJV19DIX9Zu5YGlpWzYto/+yQn88/nDuXZyNv2SteiLdG0qrtvBuKw0pgzrz/zlJdx4bo4mthcRkW6p6mAdv3t9M4+8Usb2fbUUZCTzk0vH8tnTM/WzUboNFdftZM6MAq68/zWeWlXO9efkBB1HRESkw5TvOcBDr5Ty9Kpy9h9u4Jz8fvz40rFMG56uRV+k21Fx3U7OyuvHxJw+3LekmKsnZZMQpzueRUSka3tr814eXFbKC+9sI8aMi8cP5qbzchmTmRp0NJHAqLhuR7fNKOCGh1fx7Ftb+PzEIUHHERERaXcNjc5LGz7kwWUlrN60l5TEOL4yNY8bzslhUKoWfRFRcd2Opg1PZ0xmb369pJjLzswiVpfCRESkizhwuJ5n3qjgoeWllO0+QFafntzxmVF8fuIQknuonBA5Qv8a2pGZcdv0Am554k2eW7eNS8YPDjqSiIjIKdmxr5ZHV5TxxMrNVB6oY8KQNOZdMIILRg8gTou+iHyMiut2dsHogRRkJHPPoiIuHjdIE+OLiEin9P72ah5YVsKCt7dS19jIJ0cN4CtT8jhzaB/9bBM5hoj+ymlmF5rZ+2ZWZGbfbmX/VDN708zqzezyZtsnmNkKM1tvZmvN7MpI5mxPMTHGrdPzeW97NS+/uyPoOCIiIm3m7iz9YCfXzV/JBf+zlOfWbuOqSUNY9C/Tue+6QgpztJqiyPFErOfazGKBecAngApglZktcPcNzZptBm4AvtXi8APAF919o5kNBt4wsxfdvTJSedvTJeMH88uXPmDuoiJmjczQiUhERKLaofoGFry9lfnLS3lvezXpKT341wtO45qzsknrlRB0PJFOJZLDQiYBRe5eAmBmTwKzgabi2t3Lwvsamx/o7h80e77VzHYA6UCnKK7jYmP42rR8/r9n3+HV4t2cW9A/6EgiIiIfU3ngME+s3Myjr5axo/oQpw1I4WeXj2P2hMH0iNOiLyInI5LFdSZQ3ux1BXDWib6JmU0CEoDidsrVIS4/M4u7Xt7IvEVFKq5FRCSqbNq9n/nLS/n96goO1jUwZVh/fn7FeKYO66+rrSKnKKpvaDSzQcBvgOvdvbGV/TcDNwNkZ2d3cLpjS4yP5eapefznc+/y5ua9nJHdJ+hIIiLSzb2xaQ8PLC3lxQ3biYsxZk/I5MtTchkxsHfQ0US6jEgW11uA5iupZIW3tYmZ9QaeA77r7q+11sbd7wfuBygsLPSTjxoZV0/KZt6iIuYtLGL+DRODjiMiIt1QfUMjL67/kAeXl/DW5kpSe8Zzy7R8rj8nhwG9E4OOJ9LlRLK4XgUMM7NcQkX1VcAX2nKgmSUAfwIec/dnIhcxspJ6xHHjubn88qUP2LB1H6MGq2dAREQ6Rs2hep5eVc7Dr5ZSvucg2X178YNLRnNFYRa9EqL6wrVIpxaxf13uXm9mc4AXgVjgIXdfb2Z3AqvdfYGZTSRURPcBLjazH7j7aODzwFSgn5ndEH7LG9z97UjljZTrz87h/qUl3LO4iLlfOCPoOCIi0sVtr6rlkVfLeGLlJqpr6zlzaB+++6mRfGLUQK0cLNIBIvqrq7s/DzzfYtsdzZ6vIjRcpOVxjwOPRzJbR0ntFc91Zw/l3iXFfHNnDXnpyUFHEhGRLmjD1n08uKyEBWu20ujOhWMG8uUpebrnR6SD6bpQB7jpvFwefqWUXy8u5udXjA86joiIdCGvFu9i3qIiXinaTa+EWK6dPJQvnZtLdr9eQUcT6ZZUXHeA/sk9uGpiNo+/tolvnD+MrD464YmIyKl7f3s11zy4koyUHvz7hSP4wqRsUnvFBx1LpFuL6PLn8g83T83DDO5fWhJ0FBER6SLuXriRXvGx/PUbU7ller4Ka5EooOK6gwxO68mlp2fx5KpydlTXBh1HREQ6uaIdNTy3bhvXnZ1DnyQtUS4SLVRcd6BbpudT39DI/OWlQUcREZFO7p5FRSTGxfLlKblBRxGRZlRcd6Cc/kl8ZtxgHl+xicoDh4OOIyIinVTZrv08+/YWrjkrm/7JPYKOIyLNqLjuYLfOyGf/4QYeebUs6CgiItJJ3bO4iLjYGG6emhd0FBFpQcV1BxsxsDfnjxzAw6+UUXOoPug4IiLSyZTvOcAf39zC1ROHkKHly0WijorrAMyZWUDVwTp+u3JT0FFERKSTuXdJMWbw1Wn5QUcRkVaouA7AhCFpnFfQnweWlVJb1xB0HBER6SS2V9Xy+9UVXFE4hMFpPYOOIyKtUHEdkNtmFLCz+hC/X10edBQREekk7l1STKM7t6jXWiRqqbgOyOS8vpyRnca9S0qoa2gMOo6IiES5HdW1/O71zXzu9EyG9NVKvyLRSsV1QMyMOTML2FJ5kD+/vTXoOCIiEuUeWBrqjLltRkHQUUTkGFRcB2jGaRmMGtSbexYX0dDoQccREZEotbvmEI+/tplLxg8mp39S0HFE5BhUXAfIzLhtRgElO/fz13e2Bx1HRESi1PzlpdTWNzBnpnqtRaKdiuuAXThmIHnpScxdVIS7eq9FROSjKg8c5rEVm/jU2EEUZKQEHUdEjkPFdcBiY4xbpuXz7rZ9LH5/Z9BxRKQbM7MLzex9Mysys2+3sn+omb1sZmvNbLGZZTXb9zMzW29m75rZXWZmHZu+6zqy6Njt6rUW6RRUXEeBz56eSWZaT/Vei0hgzCwWmAdcBIwCrjazUS2a/QJ4zN3HAXcCPw4few5wLjAOGANMBKZ1UPQubV9tHQ+9UsonRw1gxMDeQccRkTZQcR0F4mNj+Nq0PN7YtJfXSvYEHUdEuqdJQJG7l7j7YeBJYHaLNqOAheHni5rtdyARSAB6APHAhxFP3A089moZ1bX13D5zWNBRRKSNIlpct+ES41Qze9PM6s3s8hb7rjezjeHH9ZHMGQ2uKBxC/+QezFtUFHQUEemeMoHmq1pVhLc1twa4NPz8c0CKmfVz9xWEiu1t4ceL7v5uax9iZjeb2WozW71zp4bCHcv+Q/XMX17KjNPSGZuVGnQcEWmjiBXXbbzEuBm4Afhti2P7At8DziLUm/I9M+sTqazRIDE+lq9MyWV50S7eLq8MOo6ISGu+BUwzs7cIDfvYAjSYWQEwEsgiVJDPNLMprb2Bu9/v7oXuXpient5RuTulx1/bxN4Dddw+S73WIp1JJHuuj3uJ0d3L3H0t0HKJwguAl9x9j7vvBV4CLoxg1qhwzeShpPaMV++1iARhCzCk2eus8LYm7r7V3S9199OB74a3VRLqxX7N3WvcvQZ4ATi7Y2J3TQcPN/DAshKmDOvPGdldum9JpMuJZHHdlkuMkTi200ruEceN5+bw0oYPeW/7vqDjiEj3sgoYZma5ZpYAXAUsaN7AzPqb2ZGfG98BHgo/30yoRzvOzOIJ9Wq3OixE2ua3r29mV81hjbUW6YQ69Q2NXXHs3g3n5JCUEMs9i4qDjiIi3Yi71wNzgBcJFcZPu/t6M7vTzC4JN5sOvG9mHwADgB+Ftz8DFAPrCI3LXuPu/9eR+buS2roG7ltSzFm5fZmU2zfoOCJygiJZXB/3EuOpHtsVx+6l9Urg2slD+cvarZTt2h90HBHphMzs4mY9zG3m7s+7+3B3z3f3H4W33eHuC8LPn3H3YeE2X3b3Q+HtDe7+VXcf6e6j3P2b7fuNupffry5nR/Uhvq6x1iKdUiSL6+NeYjyGF4FPmlmf8I2Mnwxv6xZumpJLXGwM9y5R77WInJQrgY3hhV1GBB1G2u5wfSO/XlzMGdlpnJPfL+g4InISIlZct+USo5lNNLMK4ArgPjNbHz52D/BDQgX6KuDO8LZuISMlkasmDuEPb1awtfJg0HFEpJNx92uB0wkN1XjEzFaEh9Fp7ewo98c3K9haVcvXZw1Di1yKdE4RHXPdhkuMq9w9y92T3L2fu49uduxD7l4QfjwcyZzR6KvT8nGH+5eWBB1FRDohd99HaCz0k8AgQjN6vGlmtwcaTI6qrqGReYuLGJeVyrThXWOoo0h31KlvaOzKMlP8k/EAACAASURBVNN68rnTM3ly1WZ21RwKOo6IdCJmdomZ/QlYTGi1xEnufhEwHviXILPJ0f357a2U7znI7TPVay3Smam4jmJfm57PofpG5i8vDTqKiHQulwG/cvex7v5zd98B4O4HgJuCjSataWh07llUxMhBvTl/ZEbQcUTkFKi4jmL56cl8auwgfrNiE1UH64KOIyKdx/eB14+8MLOeZpYD4O4vBxNJjuUva7dSsms/t88sUK+1SCen4jrK3Ta9gJpD9Tz2alnQUUSk8/g9H135tiG8TaJQY6Mzb1ERwzKSuXD0wKDjiMgpUnEd5UYN7s2sERk89Eop+w/VBx1HRDqHOHc/fORF+HlCgHnkGF5cv50PPqxhzswCYmLUay3S2am47gRum1nA3gN1/O71zUFHEZHOYWezVRUxs9nArgDzyFG4O3ctLCKvfxKfGTc46Dgi0g5UXHcCZ2T34ey8fty/tIRD9Q1BxxGR6Pc14P+Z2WYzKwf+HfhqwJmkFX9/dwfvbtvHrTMKiFWvtUiXoOK6k5gzs4Ad1Yd45o2KoKOISJRz92J3nwyMAka6+znuXhR0Lvkod+fuhRsZ0rcnsyeo11qkq4hrSyMzSwIOunujmQ0HRgAvuLumsOgg5+T3Y8KQNO5dUsyVhUOIi9XvRSJydGb2aWA0kHhk9gl3vzPQUPIRSz7YydqKKn586VjidU4X6TLa+q95KaETdCbwN+A64JFIhZKPMzPmzCigfM9BFqzZGnQcEYliZnYvcCVwO2DAFcDQQEPJR4R6rYsYnJrIZWdkBR1HRNpRW4trCy8+cClwj7tfQahHRDrQzBEZjBiYwj2Li2ls9KDjiEj0OsfdvwjsdfcfAGcDwwPOJM2sKN7NG5v2csv0fBLi1Gst0pW0ubg2s7OBa4DnwttiIxNJjiYmxrh1RgFFO2r424btQccRkehVG/7zgJkNBuqAQQHmkRb+9+WNZKT04IrCIUFHEZF21tbi+p+A7wB/cvf1ZpYHLIpcLDmaT48dRG7/JOYuKsJdvdci0qr/M7M04OfAm0AZ8NtAE0mT10v3sLJ0D1+dlk9ivPqpRLqaNhXX7r7E3S9x95+aWQywy92/HuFs0orYGOOWafm8s2UfSz7YGXQcEYky4XP0y+5e6e5/IDTWeoS73xFwNAm7e+FG+icn8IVJ2UFHEZEIaFNxbWa/NbPe4VlD3gE2mNm/RjaaHM1nT89kcGoi8xZpZi0R+Sh3bwTmNXt9yN2rAowkzby1eS/LNu7iy1Py6JmgXmuRrqitw0JGufs+4LPAC0AuoRlDJAAJcTHcPDWPVWV7eb10T9BxRCT6vGxml9mROfgkaty9sIg+veK5brImbxHpqtpaXMebWTyh4npBeH5rDfgN0FWTsumfnMBc9V6LyMd9Ffg9cMjM9plZtZntCzpUd/fOlioWvreDm87LJalHm5aZEJFOqK3F9X2EbohJApaa2VBAJ+oAJcbHctN5eSz9YCdrKyqDjiMiUcTdU9w9xt0T3L13+HXvoHN1d3e9vJHeiXF88ZycoKOISAS19YbGu9w9090/5SGbgBkRzibHce3kbHonxmnstYh8hJlNbe0RdK7u7N1t+/jbhg+54dxceifGBx1HRCKorTc0pprZL81sdfjx34R6sY933IVm9r6ZFZnZt1vZ38PMngrvX2lmOeHt8Wb2qJmtM7N3zew7J/i9uoWUxHhuOCeHF9d/yAcfVgcdR0Six782e/wH8H/A94MM1N3NXVREUkIsXzo3J+goIhJhbR0W8hBQDXw+/NgHPHysA8wsltAd6xcBo4CrzWxUi2Y3EVpBrAD4FfDT8PYrgB7uPhY4E/jqkcJbPurGc3PplRDLrxcXBx1FRKKEu1/c7PEJYAywN+hc3VXRjmqeX7eN68/JIa1XQtBxRCTC2lpc57v799y9JPz4AZB3nGMmAUXh9oeBJ4HZLdrMBh4NP38GmBW+u92BJDOLA3oCh9EY71b1SUrgmrOyWbBmK5t3Hwg6johEpwpgZNAhuqt5i4pJjIvlpvNyg44iIh2grcX1QTM778gLMzsXOHicYzKB8mavK8LbWm3j7vVAFdCPUKG9H9gGbAZ+4e4fm3POzG4+MlRl587uu6DKV6bkERtj/HqJeq9FBMzsbjO7K/yYCywjtFKjdLCyXfv589tbuHZyNv2SewQdR0Q6QFvnAvoa8JiZpYZf7wWuj0wkINTr3QAMBvoAy8zs7+5e0ryRu98P3A9QWFjYbacGzOidyOcLs3h6VQXfmDWMgamJQUcSkWCtbva8Hvidu78SVJjubN6iIuJjY/jK1ONd7BWRrqKts4WscffxwDhgnLufDsw8zmFbgCHNXmeFt7XaJjwEJBXYDXwB+Ku717n7DuAVoLAtWburr07Np8GdB5aVHL+xiHR1zwCPu/uj7v4E8JqZ9Qo6VHdTvucAf3prC1dPyiYjRZ0eIt1FW4eFAODu+8IrNQJ88zjNVwHDzCzXzBKAq4AFLdos4B894JcDC93dCQ0FmQkQXnJ9MvDeiWTtbob07cXsCYP57crN7Nl/OOg4IhKslwndr3JET+DvAWXptn69pJgYM746Tb3WIt3JCRXXLRxzWd3wGOo5wIvAu8DT7r7ezO40s0vCzeYD/cysiFCxfmS6vnlAspmtJ1SkP+zua08ha7dw6/QCausbeGh5adBRRCRYie5ec+RF+Ll6rjvQtqqDPLO6gisKsxiU2vP4B4hIl3Eq668ed4yzuz8PPN9i2x3NntcSmnav5XE1rW2XYyvISOaiMQN5dEUZN0/L00IFIt3XfjM7w93fBDCzMzn+TejSju5bUkKjO7dMzw86ioh0sGP2XJtZtZnta+VRTehmQ4kyt04voLq2nt+s2BR0FBEJzj8BvzezZWa2HHiK0JVE6QA79tXyu9c3c+kZmWT10QUDke7mmD3X7p7SUUGkfYzJTGX6aenMX17Kl87NpWdCbNCRRKSDufsqMxsBnBbe9L671wWZqTu5f2kJdQ2N3Dq9IOgoIhKAUxlzLVFqzowC9uw/zO9e3xx0FBEJgJndBiS5+zvu/g6he1huDTpXd7C75hBPrNzM7AmZ5PRPCjqOiARAxXUXVJjTl7Ny+3L/0hIO1TcEHUdEOt5X3L3yyAt33wt8JcA83caDy0uprW/gthnqtRbprlRcd1FzZhawfV8tf3yz5dTiItINxJpZ04xOZhYLJASYp1uoPHCYx14t49NjB1GQkRx0HBEJiIrrLuq8gv6My0rl3iXF1Dc0Bh1HRDrWX4GnzGyWmc0Cfge8EHCmLu+hV8rYf7iBOTPVay3Snam47qLMjNtmFLBp9wGeW7ct6Dgi0rH+HVgIfC38WMdHF5VplZldaGbvm1mRmX27lf1DzexlM1trZovNLCu8fYaZvd3sUWtmn23n7xTV9tXW8fArpVwwegAjBvYOOo6IBEjFdRf2iZEDGD4gmXmLimhsPO605CLSRbh7I7ASKAMmEVrx9t1jHRMeOjIPuAgYBVxtZqNaNPsF8Ji7jwPuBH4c/rxF7j7B3SeEP+sA8Ld2+0KdwKOvlFFdW8/tM4cFHUVEAqbiuguLiQn1Xn/wYQ0vvfth0HFEJMLMbLiZfc/M3gPuBjYDuPsMd597nMMnAUXuXuLuh4Engdkt2owi1CMOsKiV/QCXAy+4+4GT/R6dTc2heua/UsrMERmMyUwNOo6IBEzFdRf36bGDyO7bi3mLinBX77VIF/ceoZ7jz7j7ee5+N9DWKYMygfJmryvC25pbA1wafv45IMXM+rVocxWhMd7dxuOvbaLyQB23a6y1iKDiusuLi43hlun5rK2oYnnRrqDjiEhkXQpsAxaZ2QPhmxntOMeciG8B08zsLWAasIVmxbuZDQLGAi8e7Q3M7GYzW21mq3fu3NmO0YJx8HADDy4rYcqw/pye3SfoOCISBVRcdwOXnpHJwN6JzF1YFHQUEYkgd3/W3a8CRhAatvFPQIaZ/drMPnmcw7cAQ5q9zgpva/7+W939Unc/HfhueFtlsyafB/50rNUg3f1+dy9098L09PQ2f7do9dvXN7Or5jBfn6Wx1iISouK6G+gRF8vNU/NYWbqH1WV7go4jIhHm7vvd/bfufjGhIvktQjOIHMsqYJiZ5ZpZAqHhHQuaNzCz/mZ25OfGd4CHWrzH1XSjISG1dQ3ct6SYyXl9mZjTN+g4IhIlVFx3E1dPyqZvUgJzF6n3WqQ7cfe94d7iWcdpVw/MITSk413gaXdfb2Z3mtkl4WbTgffN7ANgAPCjI8ebWQ6hnu8l7f4lotTTq8vZUX2Ir2uGEBFpJi7oANIxeibEctN5ufz8xfd5Z0uV7mgXkY9x9+eB51tsu6PZ82eAZ45ybBkfvwGyyzpc38i9i4spHNqHs/Nb3tMpIt2Zeq67kevOHkpKYhz3LFbvtYjIqfjDmxVsrarl9lnDaLbSvIiIiuvupHdiPNefncML72ynaEd10HFERDqluoZG7llcxPisVKYO6x90HBGJMiquu5kvnZdLYlws9ywuDjqKiEin9Oe3t1K+5yC3z1SvtYh8nIrrbqZvUgJXT8oO/3DoNguoiYi0i4ZGZ96iIkYN6s2skRlBxxGRKBTR4trMLjSz982syMy+3cr+Hmb2VHj/yvDd5kf2jTOzFWa23szWmVliJLN2JzdPzSPWjPuWqvdaRORE/GXtVkp37ef2mQXqtRaRVkWsuDazWGAecBEwCrjazEa1aHYTsNfdC4BfAT8NHxsHPA58zd1HE5r+6aiLEsiJGZiayGVnZvH06gp27KsNOo6ISKfQ2OjMXVjE8AHJXDB6YNBxRCRKRbLnehJQ5O4l7n4YeBKY3aLNbODR8PNngFkW6gr4JLDW3dcAuPtud29A2s0t0/JpaHQeWFYSdBQRkU7hr+u3s3FHDXNmDiMmRr3WItK6SBbXmUB5s9cVfHwO1KY24QUMqoB+wHDAzexFM3vTzP6ttQ8ws5vNbLWZrd65c2e7f4GuLLtfLy4ZP5gnVm5m7/7DQccREYlqjY3OXS9vJC89iU+PHRR0HBGJYtF6Q2MccB5wTfjPz5nZx1YXC686Vujuhenp6R2dsdO7ZXo+Bw438PCrZUFHERGJan9/90Pe217NbdMLiFWvtYgcQySL6y2ElsI9Iiu8rdU24XHWqcBuQr3cS919l7sfILRi2BkRzNotDR+QwgWjB/DIK6VU12pIu4hIa9yduxcWkd23F7MnDA46johEuUgW16uAYWaWa2YJwFXAghZtFgDXh59fDix0dwdeBMaaWa9w0T0N2BDBrN3WnBnD2Fdbz+OvbQ46iohIVFr8wU7Wbanithn5xMVG6wVfEYkWETtLhMdQzyFUKL8LPO3u683sTjO7JNxsPtDPzIqAbwLfDh+7F/gloQL9beBNd38uUlm7s7FZqUwdns785SXU1umeURGR5tydu1/eSGZaTz53elbQcUSkE4iL5Ju7+/OEhnQ033ZHs+e1wBVHOfZxQtPxSYTdNj2fK+9/jSdf38wN5+YGHUdEJGq8WrybNzdX8sPPjiEhTr3WInJ8OlMIZ+X1Y2JOH+5fWsLh+sag44iIRI27Xt7IgN49uOJM9VqLSNuouBYAbptRwNaqWp59q+U9pyIi3dPKkt2sLN3DV6fmkxgfG3QcEekkVFwLANOGpzM2M5VfLymmodGDjiMiEri7FxbRPzmBqydlBx1FRDoRFdcCgJlx24x8Snft57l124KOIyISqDc372V50S5unppHzwT1WotI26m4liafHDWQgoxk7llURF2Dxl6LSPd198sb6dMrnmvOGhp0FBHpZFRcS5OYGOP2mQW8t72ac3+ykF++9AHbqg4GHUtEpEOtq6hi0fs7+fKUPJJ6RHRSLRHpgnTWkI+YPSGT3onxPLaijLsXbmTuwo3MGjmAaycPZUpBf2K07K+IdHF3L9xI78Q4vni2eq1F5MSpuJaPmTEigxkjMijfc4Dfvb6Zp1eX89KGD8nu24svnJXNFWdm0S+5R9AxRUTa3bvb9vG3DR/yjVnDSEmMDzqOiHRCGhYiRzWkby/+7cIRvPrtWdx19ekMSk3kJy+8x9k/Xsg/PfkWq8r2EFqtXkSka5i7sIjkHnF8SQtqichJUs+1HFdCXAyXjB/MJeMHs/HDap5YuZk/vFHBs29v5bQBKVw7OZvPnp6pXh4R6dSKdlTz/DvbuHV6Pqm9dD4TkZOjnms5IcMGpPD9S0az8ruz+OllY0mIi+E//ryes/7rZb7zx3Ws31oVdEQRkZMyd2ERPeNjuem8vKCjiEgnpp5rOSm9EuK4cmI2V07MZk15JU+s3MSf3qrgd69v5vTsNK49ayifHjdIq5qJSKdQums/C9Zs5ctT8uiblBB0HBHpxNRzLads/JA0fnb5eFZ+53zu+Mwo9h2s419+v4bJP36Z//zLBkp37Q86oojIMc1bVER8bAxfnqKx1iJyatRzLe0mtVc8XzovlxvPzeG1kj08vnITj7xaxoPLSzm3oB/XnjWU80cNID5Wv9OJSPQo33OAP721hesmDyUjJTHoOCLSyam4lnZnZpyd34+z8/uxo7qWp1eV87vXy7nliTfJSOnBVROHcNWkbAan9Qw6qogI9ywuJtaMr03LDzqKiHQBKq4lojJSEpkzcxi3TC9g8fs7ePy1Tdy9qIi5i4qYNXIA15yVzdRh6VqcRkQCsbXyIM+8Uc6VE4cwMFW91iJy6lRcS4eIjTFmjRzArJEDtDiNiESN+5YU4456rUWk3Wjwq3S4Yy1O8w0tTiMiHWTHvlp+t6qcy87IIqtPr6DjiEgXoZ5rCczRFqf5sxanEZEOcN/SEhoanVtnqNdaRNpPRHuuzexCM3vfzIrM7Nut7O9hZk+F9680s5wW+7PNrMbMvhXJnBI8LU4jIh1pV80hnli5idkTBjO0X1LQcUSkC4lYz7WZxQLzgE8AFcAqM1vg7huaNbsJ2OvuBWZ2FfBT4Mpm+38JvBCpjBJ9jrU4zYQhaVw7eSif0eI0InKKHlxWyqH6Rm6bURB0FBHpYiLZcz0JKHL3Enc/DDwJzG7RZjbwaPj5M8AsMzMAM/ssUAqsj2BGiWItF6eprq3jW79fw1n/9TI//MsGSnbWBB1RRDqhvfsP85sVZXxm3GDy05ODjiMiXUwkx1xnAuXNXlcAZx2tjbvXm1kV0M/MaoF/J9TrfdQhIWZ2M3AzQHZ2dvsll6jS2uI0j75axnwtTiMiJ+HhV0rZf7iBOeq1FpEIiNYbGr8P/Mrda8Id2a1y9/uB+wEKCws1vUQXp8VpRORUVR2s4+FXy7hw9EBOG5gSdBwR6YIiWVxvAYY0e50V3tZamwoziwNSgd2EergvN7OfAWlAo5nVuvvcCOaVTuRYi9PMHDGAaydrcRoR+bhHXy2juraeOTPVay0ikRHJ4noVMMzMcgkV0VcBX2jRZgFwPbACuBxY6KEJjqccaWBm3wdqVFhLa462OM3f39XiNCInyswuBP4XiAUedPeftNg/FHgISAf2ANe6e0V4XzbwIKEOEwc+5e5lHZf++GoO1fPQK6WcPzKDMZmpQccRkS4qYoNU3b0emAO8CLwLPO3u683sTjO7JNxsPqEx1kXAN4GPTdcn0lZanEbk5DWb4ekiYBRwtZmNatHsF8Bj7j4OuBP4cbN9jwE/d/eRhG5o3xH51CfmNys2UXmgjttnDgs6ioh0YREdc+3uzwPPt9h2R7PntcAVx3mP70cknHRZx1uc5prJ2XxOi9OItNQ0wxOAmR2Z4an59KmjCHWEACwCng23HQXEuftLAO4edVP5HDhcz4PLSpg6PJ3xQ9KCjiMiXZimV5AurbXFae5otjjNO1u0OI1IWGszPGW2aLMGuDT8/HNAipn1A4YDlWb2RzN7y8x+Hu4Jjxq/XbmZ3fsP83WNtRaRCIvW2UJE2pUWpxFpF98C5prZDcBSQvfTNBD6WTIFOB3YDDwF3EBo6N9HBDGFam1dA/ctLeHsvH4U5vTtkM8Uke5LPdfS7WhxGpFWHXeGJ3ff6u6XuvvpwHfD2yoJ9XK/HV40rJ7QcJEzWvsQd7/f3QvdvTA9PT0S3+NjnlpVzs7qQ3x9lsZai0jkqedaui0tTiPyEced4cnM+gN73L0R+A6hmUOOHJtmZunuvhOYCazusOTHcKi+gXuXFDMxpw+T89RrLSKRp+Jauj0tTiPStErukRmeYoGHjszwBKx29wXAdODHZuaEhoXcFj62wcy+BbxsoZW/3gAeCOJ7tPSHN7awraqWn142jmMtSiYi0l6sq0xNVlhY6KtXR0VHiXQBDY3etDjN4g92YsDwASlMGJLG+CFpjM9KY/iAZOLUqy3txMzecPfCoHN0pEift+saGpnxi8X0S+7Bs7eeo+JaRNrNsc7Z6rkWaUXLxWn+9NYW3ti0l7+u386Tq0ITKiTGxzA2M5XxWWlMyA4V3Fl9euoHuEiUePatLVTsPcgPLhmtf5ci0mFUXIscx5C+vZpuhHJ3Nu85wNvllbxdXsma8koee20TDy4vBaBfUkJTz/b4IaHCu09SQpDxRbql+oZG5i0qYvTg3swckRF0HBHpRlRci5wAM2NovySG9kti9oTQFMB1DY28v726qdheU1HJovd3cGTE1dB+vULDSbJCQ0pGD+6tKf9EIuwva7dRtvsA9157pnqtRaRDqbgWOUXxsTGMyUxlTGYq104eCkDNoXrWVVQ1Fdyvl+7hz29vBSAuxhgxKKWp2D59SBp56cnExqgAEGkPjY3O3EVFnDYghU+OGhB0HBHpZlRci0RAco+4phlIjvhwX21Tz/aa8ioWvL2VJ1Zubmo/NjOV8UPSmDAk9OfA3onqcRM5CS+8s52iHTXcffXpxOiXVhHpYCquRTrIgN6JfHL0QD45eiAQ6l0r2bW/WcFdyfzlJdQ1hMaTZKT0CBfbocfYrFR6J8YH+RVEol5jo3P3wo3kpSfxqbGDgo4jIt2QimuRgMTEGAUZyRRkJHPZmVlAaMGLd7dVs6bZDZMvbfiw6Zj89KSmgnt8VhojB/UmIU7TAYoc8fd3P+S97dX88vPjNdRKRAKh4lokivSIi23qqb4+vK3qQB1rt1SGC+4qln6wiz++GVqVOiE2hlGDe4fn3w7NTpLTL0mXwqVbcnfuWriRof16ccn4wUHHEZFuSsW1SJRL7RXPlGHpTBmWDoQKiG1VtU2922+XV/L06nIeebUMgN6JcU3TAR5Z9CY9pUeA30CkYyx+fyfvbNnHzy4bpwWeRCQwKq5FOhkzY3BaTwan9eSi8JjShkanaEdNqOAOj9/+9ZJiGhpD47cz03o29WyPH5LG2MxUknron790HUd6rTPTevK5MzKDjiMi3Zh+uop0AbExxmkDUzhtYAqfnzgEgIOHG1i/NTwdYEUVa8oreX7ddgBiLLSc+5Fie/yQVE4bkKLePum0XinazVubK/nPz44hXv8fi0iAVFyLdFE9E2IpzOlLYU7fpm179h9umplkTXklf9uwnadW/2M59zGDU8PFdmj+bS3nLp3FXQs3MrB3IlcUZgUdRUS6ORXXIt1I36QEZpyWwYzTQstBuzvlew42DSVZU17J469tYn54Ofe+SQmMz/pHwT0+K42+Ws5dosxrJbt5vXQP37t4FD3itPqpiAQrosW1mV0I/C8QCzzo7j9psb8H8BhwJrAbuNLdy8zsE8BPgATgMPCv7r4wkllFuiMzI7tfL7Kbza5Q19DIBx9Ws6a8qummycUfbGxazj27b6+m6QBHDEwhLz1JC95IoO5euJH+yT24elJ20FFERCJXXJtZLDAP+ARQAawyswXuvqFZs5uAve5eYGZXAT8FrgR2ARe7+1YzGwO8COgOFZEOEB8bw+jBqYwenMoXzgoVKzWH6nlnS1XTgjdvbtrL/63Z2nRMr4RYcvsnkZeeTF7/JPLSk8jrn0xeepJunJSIemPTXl4p2s13PzWSxHj1WotI8CL5U28SUOTuJQBm9iQwG2heXM8Gvh9+/gww18zM3d9q1mY90NPMerj7oQjmFZGjSO4Rx+S8fkzO+8dy7jurD7FxRzUlO/eHHrtCs5U8t3Yr4UlK/v/27jRIjroO4/j3yV7Zi4XsbsWYg00CsRTlMiIYDwoVLbHgBVhGUS7xQFHUKhR9gaXFG31hWQGqLFQsPMEDqIBBRRJvRBCSQAKazRIlGMluMCGXIZv8fDGdzexmNpksPT29M8+nais9Pf9MP9tJ//a3Pf/uAWD6MS0jjfa83sKf83s6mHlcqz/kw16yG5evY1p7Mxef6bPWZpYPlWyuZwLPFD3eCLx+vDERMSxpG9BN4cz1ARcCj5ZqrCV9BPgIwJw5LqxmWertbKG3s4U3zO8ZtX7P8D7+uWUXA4M7WF/UeN+7ehPbdu8dGdfcOIW+7raSjXdXmz/m3Y5s9cat/Pbvg1z7jlfQ1ux3SMwsH3JdjSSdRGGqyLmlno+IW4BbABYuXBilxphZtloaG1gwvZMF0ztHrY8Int/5IgNDOxkY3MHA4E7WD+5k3ebt/ObJ5xguOt3d3d48amrJgSknx3e3+TZrNuLG5f10tTZxyVnHVzuKmdmISjbXzwKzix7PStaVGrNRUiPQReHCRiTNAu4CLomI9RXMaWYZkER3RwvdHS28ruj2gADD+/bzzH93jzTdA0OFs94PPLWZOx45+KZVwxQxZ1rbwXndI3O8O+jpaPZFlXVk7b9f4P61z/GZty2gc6rf6TCz/Khkc/0wcKKkuRSa6MXA+8eMWQpcCjwIXAQsj4iQdCzwC+C6iPhTBTOaWQ40Nkxhbk/hDPVbXzn6uW279/J00dnugaHCn3/sH2LP8P6RcZ1TG5nX28H84sa7t52+7nZf6FaDblqxjs6WRi5b1FftKGZmo1SsuU7mUF9N4U4fDcCtEbFG0leARyJiKfAd4PuS+oHnKTTgNJ909gAACT5JREFUAFcDJwDXS7o+WXduRGyuVF4zy6eu1iZOTW79V2z//uDZrbtHTTN5emgnfxnYwp2PHXyTTCp8/HupO5nM6PItBCejdc9t574n/sMnzj6BrlaftTazfKnonOuIWAYsG7Pu+qLl/wHvKfH3bgBuqGQ2M5vcpkwRs6e1MXtaG29Z0DvquV0vDidnu3eOOtv90w3Ps/PFfSPjWpsO3EKwcKZ7ftJ4z+1tp8O3EMytm1b009rUwBVvnFvtKGZmh/BPDzOrOW3NjSP36i4WEWzevof1B6aYJI336o3bWPb4Jt9CcBJ4emgn96z6Nx9+0zx/WqiZ5ZKbazOrG5KYfsxUph8ztaxbCD49tINfPL6JrbtK30Jwbm/7yAWV83vbObbNzV6l3byin+bGKVz5pnnVjmJmVpKbazMzxr+FIFC4heCB2wcmU0zWbd7OA089x959B093T2tv5u2vnM5XLzo5y+h1419bdnHXY89y6Vl99Ha2VDuOmVlJbq7NzI5gWnsz09qnsbCMWwi+vKu1Silr3/+G97HohB4++haftTaz/HJzbWY2QYe7haClb8H0Tr53xRnVjmFmdlj+qDMzMzMzs5S4uTYzMzMzS4mbazMzMzOzlLi5NjMzMzNLiZtrMzMzM7OUuLk2MzMzM0uJm2szMzMzs5S4uTYzMzMzS4ki4sijJgFJg8A/J/BXe4ChlONMVF6y5CUHOEspeckB+cmSlxww8SzHR0Rv2mHyrAbqdl5yQH6y5CUH5CdLXnKAs5SSes2umeZ6oiQ9EhELq50D8pMlLznAWfKcA/KTJS85IF9ZalVe9nFeckB+suQlB+QnS15ygLNklcPTQszMzMzMUuLm2szMzMwsJW6u4ZZqByiSlyx5yQHOUkpeckB+suQlB+QrS63Kyz7OSw7IT5a85ID8ZMlLDnCWUlLPUfdzrs3MzMzM0uIz12ZmZmZmKXFzbWZmZmaWkrppriW9U9LfJfVLuq7E8y2S7kief0hSXxWzXCZpUNLK5OvKCuW4VdJmSU+M87wkLUlyrpZ0epVynC1pW9H+uL5COWZLWiFpraQ1kq4pMSarfVJOlqz2y1RJf5W0Ksny5RJjKn78lJkjk2OnaHsNkh6TdG+J5zKrKbXINbtkDtfsQ7eVi7rtmj3hHLVZsyOi5r+ABmA9MA9oBlYBrxoz5uPAN5PlxcAdVcxyGXBTBvvlzcDpwBPjPP8u4D5AwJnAQ1XKcTZwbwb7YwZwerLcCfyjxL9NVvuknCxZ7RcBHclyE/AQcOaYMRU/fsrMkcmxU7S9zwI/KvXvkFVNqcUv1+xxs7hmH7qtXNRt1+wJ56jJml0vZ67PAPojYiAiXgRuBy4YM+YC4LZk+WfAWyWpSlkyERG/B54/zJALgO9FwV+AYyXNqEKOTETEpoh4NFneDjwJzBwzLKt9Uk6WTCTf647kYVPyNfZK6IofP2XmyIykWcB5wLfHGZJVTalFrtkluGYfKi912zV7wjkyk2XNrpfmeibwTNHjjRz6n35kTEQMA9uA7iplAbgwefvqZ5JmVyBHOcrNmoWzkreW7pN0UqU3lrwddBqF37SLZb5PDpMFMtovyVtpK4HNwP0RMe5+qeTxU0YOyO7Y+QbwOWD/OM9nVVNqkWv2xNRtzYb81G3X7KPKATVYs+uluZ5s7gH6IuJk4H4O/iZVrx4Fjo+IU4AbgbsruTFJHcDPgU9HxAuV3NZLzJLZfomIfRFxKjALOEPSqyu1rZeYI5NjR9K7gc0R8bdKvL5NOq7Zo2VasyE/dds1+6hz1GTNrpfm+lmg+LehWcm6kmMkNQJdwJZqZImILRGxJ3n4beC1FchRjnL2W8VFxAsH3lqKiGVAk6SeSmxLUhOFwvjDiLizxJDM9smRsmS5X4q2uRVYAbxzzFNZHT+HzZHhsbMIOF/SBgrTBM6R9IMxYzLdJzXGNXti6q5mQ37qtmv20eeo1ZpdL831w8CJkuZKaqYwUX3pmDFLgUuT5YuA5RFRiblBR8wyZi7Y+RTmblXDUuASFZwJbIuITVmHkPSyA/OeJJ1B4f9t6kUg2cZ3gCcj4uvjDMtkn5STJcP90ivp2GS5FXg78NSYYRU/fsrJkdWxExFfiIhZEdFH4RheHhEfGDMsq5pSi1yzJ6auanby+rmo267ZE8tRqzW7ccJJJ5GIGJZ0NfArCld+3xoRayR9BXgkIpZSOCi+L6mfwoUai6uY5VOSzgeGkyyXVSKLpB9TuHq5R9JG4EsULjggIr4JLKNwlXU/sAu4vEo5LgKukjQM7AYWV+iH6CLgg8DjyRwxgC8Cc4qyZLJPysyS1X6ZAdwmqYHCD4OfRMS9VTh+ysmRybEznmrUlFrkml2aa3ZJeanbrtkTy1GTNdsff25mZmZmlpJ6mRZiZmZmZlZxbq7NzMzMzFLi5trMzMzMLCVurs3MzMzMUuLm2szMzMwsJW6urS5I2idpZdHXdSm+dp+kJ9J6PTOzeueabZNZXdzn2gzYnXwEq5mZ5Z9rtk1aPnNtdU3SBklfk/S4pL9KOiFZ3ydpuaTVkh6QNCdZP13SXZJWJV9vSF6qQdK3JK2R9Ovk06iQ9ClJa5PXub1K36aZWU1wzbbJwM211YvWMW8xvrfouW0R8RrgJuAbybobgdsi4mTgh8CSZP0S4HcRcQpwOrAmWX8icHNEnARsBS5M1l8HnJa8zscq9c2ZmdUY12ybtPwJjVYXJO2IiI4S6zcA50TEgKQm4D8R0S1pCJgREXuT9ZsiokfSIDArIvYUvUYfcH9EnJg8/jzQFBE3SPolsAO4G7g7InZU+Fs1M5v0XLNtMvOZazOIcZaPxp6i5X0cvJ7hPOBmCmdMHpbk6xzMzF4a12zLNTfXZvDeoj8fTJb/DCxOli8G/pAsPwBcBSCpQVLXeC8qaQowOyJWAJ8HuoBDzsSYmdlRcc22XPNvZFYvWiWtLHr8y4g4cGun4yStpnAm433Juk8C35V0LTAIXJ6svwa4RdKHKJztuArYNM42G4AfJMVcwJKI2Jrad2RmVrtcs23S8pxrq2vJ/L2FETFU7SxmZnZ4rtk2GXhaiJmZmZlZSnzm2szMzMwsJT5zbWZmZmaWEjfXZmZmZmYpcXNtZmZmZpYSN9dmZmZmZilxc21mZmZmlpL/A8DgqIltDQ/3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axs = plt.subplots(1, 2, figsize=(12, 4))\n",
    "\n",
    "axs[0].plot(history.history['loss'])\n",
    "axs[0].set_title('Training loss')\n",
    "axs[0].set(xlabel='Epochs', ylabel='Loss')\n",
    "\n",
    "axs[1].plot(history.history['acc'])\n",
    "axs[1].set_title('Training accuracy')\n",
    "axs[1].set(xlabel='Epochs', ylabel='Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the trained model on the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 42us/step\n",
      "Test loss: 0.05459650097906269\n",
      "Test accuracy: 0.9844\n"
     ]
    }
   ],
   "source": [
    "loss, accuracy = model.evaluate(x_test,y_test)\n",
    "print(\"Test loss: {}\".format(loss))\n",
    "print(\"Test accuracy: {}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save checkpoint\n",
    "We use the [checkpoint saving method for the Vitis AI flow](https://github.com/Xilinx/Vitis-AI-Tutorials/tree/Keras-Freeze-with-Vitis-AI). We can get the session and graph from the `keras.backend` module\n",
    "and use those to save a Tensorflow checkpoint compatible with the Vitis AI\n",
    "workflow. The following cell does the following:\n",
    "\n",
    "1. Set up Tensorflow saver object\n",
    "2. Fetch the Tensorflow session using the Keras backend\n",
    "3. Get the Tensorflow session graph\n",
    "4. Write out Tensorflow checkpoint and inference graph for use with the freeze_graph script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./mnist_classifier.pb'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "saver = tf.train.Saver()\n",
    "tf_session = keras.backend.get_session()\n",
    "input_graph_def = tf_session.graph.as_graph_def()\n",
    "save_path = saver.save(tf_session, './checkpoint.ckpt')\n",
    "tf.train.write_graph(input_graph_def,\n",
    "                     './', 'mnist_classifier.pb', as_text=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As well as saving the checkpoint we also need to make a note of the input \n",
    "and output nodes of the graph for freezing and quantization. We made our\n",
    "lives a bit easier by naming the input and output layers, which results\n",
    "in our input and output nodes being named `input_data` and \n",
    "`output_logits/Softmax` respectively. You can check the node names in the \n",
    "list defined below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "nodes_names = [node.name for node in \n",
    "               tf.get_default_graph().as_graph_def().node]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Freeze Tensorflow graph\n",
    "The Vitis AI flow requires a frozen model for quantization. We can obtain a binary\n",
    "protobuf file of our frozen model by using the Tensorflow `freeze_graph` utility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "W0430 16:11:35.735119 140327132825408 deprecation.py:323] From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "2020-04-30 16:11:35.799984: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n",
      "2020-04-30 16:11:35.827766: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2600000000 Hz\n",
      "2020-04-30 16:11:35.830744: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55be91a92a50 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2020-04-30 16:11:35.830758: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "INFO:tensorflow:Restoring parameters from checkpoint.ckpt\n",
      "I0430 16:11:35.868305 140327132825408 saver.py:1284] Restoring parameters from checkpoint.ckpt\n",
      "WARNING:tensorflow:From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
      "W0430 16:11:35.922450 140327132825408 deprecation.py:323] From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.convert_variables_to_constants`\n",
      "WARNING:tensorflow:From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
      "W0430 16:11:35.922661 140327132825408 deprecation.py:323] From /opt/vitis_ai/conda/envs/vitis-ai-tensorflow/lib/python3.6/site-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.compat.v1.graph_util.extract_sub_graph`\n",
      "INFO:tensorflow:Froze 6 variables.\n",
      "I0430 16:11:35.940265 140327132825408 graph_util_impl.py:334] Froze 6 variables.\n",
      "INFO:tensorflow:Converted 6 variables to const ops.\n",
      "I0430 16:11:35.990261 140327132825408 graph_util_impl.py:394] Converted 6 variables to const ops.\n"
     ]
    }
   ],
   "source": [
    "!freeze_graph \\\n",
    "    --input_graph mnist_classifier.pb \\\n",
    "    --input_checkpoint checkpoint.ckpt \\\n",
    "    --input_binary true \\\n",
    "    --output_graph frozen.pb \\\n",
    "    --output_node_names output_logits/Softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quantization\n",
    "We will save some of our training data as calibration data for\n",
    "`vai_q_tensorflow`, then use that along with the frozen graph to quantize our model.\n",
    "\n",
    "`vai_q_tensorflow inspect` can be used to confirm available input and output\n",
    "node names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Op types used: 11 Const, 6 Identity, 3 BiasAdd, 2 MatMul, 2 Relu, 1 Conv2D, 1 MaxPool, 1 Pack, 1 Placeholder, 1 Prod, 1 Reshape, 1 Shape, 1 Softmax, 1 StridedSlice\r\n",
      "\r\n",
      "Found 1 possible inputs: (name=input_data, type=float(1), shape=[?,28,28,1]) \r\n",
      "Found 1 possible outputs: (name=output_logits/Softmax, op=Softmax) \r\n"
     ]
    }
   ],
   "source": [
    "!vai_q_tensorflow inspect --input_frozen_graph=frozen.pb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will save a portion of our training data for quantization.\n",
    "Recommended number is around 100-1000 for images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.savez('./calib_data.npz', data = x_train[:1024])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The quantizer requires an input function to feed batches of calibration data.\n",
    "In order to replicate the input in the way that the network sees during training, \n",
    "you could add normalization and other pre-processing steps here as well. \n",
    "Since we saved our calibration data already normalized, we can keep our \n",
    "function simple and just use it to pass batches of data to our model.\n",
    "\n",
    "In the next cell, we will use IPython magic to write a Python file\n",
    "`input_func.py` in the current working directory. This is required by the\n",
    "`vai_q_tensorflow` compiler as you can see later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting input_func.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile input_func.py\n",
    "import numpy as np\n",
    "\n",
    "data = np.load('calib_data.npz')['data']\n",
    "\n",
    "batch_size=32\n",
    "\n",
    "def calib_input(iter):\n",
    "\n",
    "    calib_data = data[iter*batch_size:(iter+1)*batch_size]\n",
    "\n",
    "    return {'input_data': calib_data}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Checking Float Graph...\n",
      "INFO: Float Graph Check Done.\n",
      "INFO: Calibrating for 32 iterations...\n",
      "100% (32 of 32) |########################| Elapsed Time: 0:00:02 Time:  0:00:02\n",
      "INFO: Calibration Done.\n",
      "INFO: Generating Deploy Model...\n",
      "INFO: Deploy Model Generated.\n",
      "********************* Quantization Summary *********************      \n",
      "INFO: Output:       \n",
      "  quantize_eval_model: quantized/quantize_eval_model.pb       \n",
      "  deploy_model: quantized/deploy_model.pb\n"
     ]
    }
   ],
   "source": [
    "!vai_q_tensorflow quantize \\\n",
    "    --input_frozen_graph frozen.pb \\\n",
    "    --input_fn input_func.calib_input \\\n",
    "    --output_dir quantized \\\n",
    "    --input_nodes input_data \\\n",
    "    --output_nodes output_logits/Softmax \\\n",
    "    --input_shapes ?,28,28,1 \\\n",
    "    --calib_iter 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate quantized model\n",
    "The quantizer produces a special model called `quantize_eval_model.pb`, \n",
    "which we can use to load up like a regular Tensorfow binary graph and \n",
    "evaluate its performance.\n",
    "\n",
    "Because we already have a graph definition in our session, we need to\n",
    "reset the default graph so as to not interfere with the graph we are \n",
    "about to load from the frozen model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to evaluate a quantized model we have to import `tensorflow.contrib.decent_q`, \n",
    "otherwise the model evaluation will error out. We will use standard \n",
    "Tensorflow 1, to set up the graph for evaluation.\n",
    "\n",
    "In the next cell, we will read in a frozen binary graph and add the \n",
    "accuracy metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow.contrib.decent_q\n",
    "\n",
    "with tf.gfile.GFile('quantized/quantize_eval_model.pb', \"rb\") as f:\n",
    "    graph = tf.GraphDef()\n",
    "    graph.ParseFromString(f.read())\n",
    "\n",
    "tf.import_graph_def(graph,name = '')\n",
    "\n",
    "input_data = tf.get_default_graph().get_tensor_by_name('input_data'+':0')\n",
    "labels = tf.placeholder(tf.int64, shape=[None,])\n",
    "logits = tf.get_default_graph().get_tensor_by_name(\n",
    "    'output_logits/Softmax'+':0')\n",
    "\n",
    "nn_output = tf.argmax(logits, 1)\n",
    "correct_prediction = tf.equal(nn_output, labels)\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run a Tensorflow session to evaluate the test accuracy of the quantized model.\n",
    "If the accuracy at this stage is acceptable, we can proceed to compilation \n",
    "and deployment on DPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average accuracy on test set: 0.9843000173568726\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    sess.run(tf.initializers.local_variables())\n",
    "    \n",
    "    acc = accuracy.eval(feed_dict={input_data: x_test, labels: y_test})\n",
    "    print(\"Average accuracy on test set: {}\".format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compilation\n",
    "Now that we are satisfied with our quantized model accuracy, we can compile it and move onto the target.\n",
    "\n",
    "This example targets the ZCU111 (or ZCU102 which uses the same DPU \n",
    "configuration as ZCU102). You can target a different architecture by\n",
    "specifying its configuration json file with the `--arch` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**************************************************\n",
      "* VITIS_AI Compilation - Xilinx Inc.\n",
      "**************************************************\n",
      "\u001b[0;33m[VAI_C][Warning] layer [output_logits_Softmax] (type: Softmax) is not supported in DPU, deploy it in CPU instead.\n",
      "\u001b[m\n",
      "Kernel topology \"mnist_classifier_kernel_graph.jpg\" for network \"mnist_classifier\"\n",
      "kernel list info for network \"mnist_classifier\"\n",
      "                               Kernel ID : Name\n",
      "                                       0 : mnist_classifier_0\n",
      "                                       1 : mnist_classifier_1\n",
      "\n",
      "                             Kernel Name : mnist_classifier_0\n",
      "--------------------------------------------------------------------------------\n",
      "                             Kernel Type : DPUKernel\n",
      "                               Code Size : 4.12KB\n",
      "                              Param Size : 0.66MB\n",
      "                           Workload MACs : 1.78MOPS\n",
      "                         IO Memory Space : 6.17KB\n",
      "                              Mean Value : 0, 0, 0, \n",
      "                      Total Tensor Count : 4\n",
      "                Boundary Input Tensor(s)   (H*W*C)\n",
      "                         input_data:0(0) : 28*28*1\n",
      "\n",
      "               Boundary Output Tensor(s)   (H*W*C)\n",
      "               output_logits_MatMul:0(0) : 1*1*10\n",
      "\n",
      "                        Total Node Count : 3\n",
      "                           Input Node(s)   (H*W*C)\n",
      "                 conv2d_1_convolution(0) : 28*28*1\n",
      "\n",
      "                          Output Node(s)   (H*W*C)\n",
      "                 output_logits_MatMul(0) : 1*1*10\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "                             Kernel Name : mnist_classifier_1\n",
      "--------------------------------------------------------------------------------\n",
      "                             Kernel Type : CPUKernel\n",
      "                Boundary Input Tensor(s)   (H*W*C)\n",
      "              output_logits_Softmax:0(0) : 1*1*10\n",
      "\n",
      "               Boundary Output Tensor(s)   (H*W*C)\n",
      "              output_logits_Softmax:0(0) : 1*1*10\n",
      "\n",
      "                           Input Node(s)   (H*W*C)\n",
      "                   output_logits_Softmax : 1*1*10\n",
      "\n",
      "                          Output Node(s)   (H*W*C)\n",
      "                   output_logits_Softmax : 1*1*10\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!vai_c_tensorflow \\\n",
    "    --frozen_pb quantized/deploy_model.pb \\\n",
    "    --arch /opt/vitis_ai/compiler/arch/dpuv2/ZCU102/ZCU102.json \\\n",
    "    --output_dir . \\\n",
    "    --net_name mnist_classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that some node names are different now, take a note of these, as they are required by the DNNDK API on the target. \n",
    "\n",
    "* Input node: `conv2d_1_convolution`\n",
    "* Output node: `output_logits_MatMul`\n",
    "* Kernel name: `mnist_classifier_0`\n",
    "\n",
    "Copyright (C) 2020 Xilinx, Inc"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
